Andy is one of the most influential minds in data science with a CV to match. He shares his thoughts on distributed machine learning with open-source tools like Dask-ML as well as proprietary tools from the big cloud providers. In a past post, we covered distributed ML use cases and discussed whether or not we really need distributed machine learning. You can check it out here. This interview was lightly edited for clarity.

The Cloudera Support Organization has always strived to not only provide solutions to our customers but to also deliver helpful knowledge. One of the primary sources of that knowledge comes from our Knowledge Articles. This content is created and curated by our knowledgeable Support Staff based on real-world experience coming from support cases. These Knowledge Articles have proven to be invaluable to our Support Staff over the years. While the content is also available to our customers to use in their own troubleshooting efforts, we want to do more to help bring the right Knowledge Articles to our customers at the right time.

Do you want to learn to analyze your big data using open source technologies? This introductory Spark ML workshop can help you! We will discuss many typical Machine Learning operations and show you how to perform them in Apache Spark using Python and Java API, so that you can tame your big data challenges while using your favorite programming language. Svetlana Levitan, PhD, has been a software engineer implementing statistical and Machine Learning algorithms for SPSS Inc. and IBM for 18 years and a Senior Developer Advocate at IBM more recently. Currently she is working as a Senior Data Scientist for a small startup and planning to join her next company as a Principal Data Scientist.

At Ibotta we train a lot of machine learning models. They make predictions for millions of users as they interact with our mobile app. While we do much of our data processing with Spark, our preferred machine learning framework is scikit-learn. As compute gets cheaper and time to market for machine learning solutions becomes more critical, we've explored options for speeding up model training. One of those solutions is to combine elements from Spark and scikit-learn into our own hybrid solution.

At Ibotta we train a lot of machine learning models. They make predictions for millions of users as they interact with our mobile app. While we do much of our data processing with Spark, our preferred machine learning framework is scikit-learn. As compute gets cheaper and time to market for machine learning solutions becomes more critical, we've explored options for speeding up model training. One of those solutions is to combine elements from Spark and scikit-learn into our own hybrid solution.

There are many machine learning frameworks. Given that each takes much time to learn, and given that some have a wider user base than others, which one should you use? Here we look briefly at some of the major ones. In picking a tool, you need to ask what is your goal: machine learning or deep learning? Deep learning has come to mean using neural networks to do, for the most part it seems, image recognition.

Our second core requirement was frictionless reuse of existing Spark libraries. Part of it is our own pet peeve – why does every NLP library out there have to build its own topic modeling and word embedding implementations? The other part is pragmatic – we're a small team under tight deadlines and need to make the most of what's already there. When we started thinking about a Spark NLP library, we first asked Databricks to point us to whoever is already building one. When the answer came there there isn't one, the next ask was to help us make sure the design and API of the library fully meet Spark ML's API guidelines.

Our second core requirement was frictionless reuse of existing Spark libraries. Part of it is our own pet peeve – why does every NLP library out there have to build its own topic modeling and word embedding implementations? The other part is pragmatic – we're a small team under tight deadlines and need to make the most of what's already there. When we started thinking about a Spark NLP library, we first asked Databricks to point us to whoever is already building one. When the answer came there there isn't one, the next ask was to help us make sure the design and API of the library fully meet Spark ML's API guidelines.

Then we use another map transformation, which will apply the ParseObs function to transform each Array of Double in the RDD into an Array of Cancer Observation objects. The toDF() method transforms the RDD of Array[[Cancer Observation]] into a Dataframe with the Cancer Observation class schema. Below the data is split into a training data set and a test data set, 70% of the data is used to train the model, and 30% will be used for testing. In this blog post, we showed you how to get started using Apache Spark's machine learning Logistic Regression for classification.

If you generate machine learning (ML) models, you know that the key challenge is exporting and importing them into other frameworks to separate model generation and prediction. Many applications use PMML (Predictive Model Markup Language) to move ML models from one framework to another. PMML is an XML representation of a data mining model. In this post, I show how to build a PMML application on AWS. First, you build a PMML model in Apache Spark using Amazon EMR.